feat(low-code): add RateLimitedMultipleTokenAuthenticator component#1066
Draft
Daryna Ishchenko (darynaishchenko) wants to merge 8 commits into
Draft
Conversation
… component Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1783347505-rate-limited-multiple-token-authenticator#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1783347505-rate-limited-multiple-token-authenticatorPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
PyTest Results (Full)4 142 tests 4 130 ✅ 12m 20s ⏱️ Results for commit f300579. ♻️ This comment has been updated with latest results. |
…on wait Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
1 task
… already converted Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new declarative authenticator,
RateLimitedMultipleTokenAuthenticator, that generalizes source-github'sMultipleTokenAuthenticatorWithRateLimiterso it can be reused by any connector during declarative migrations (context: https://github.com/airbytehq/airbyte-internal-issues/issues/16512).Behavior (per outgoing request, via
AuthBase.__call__):quotas) viaHttpRequestRegexMatchers; a pool with no matchers is the default (generalizes GitHub's hardcoded"graphql" in pathREST-vs-GraphQL split).quota_status_source(e.g. GitHubGET /rate_limit) usingremaining_path/reset_path/limit_pathdpath extraction. Seeding is lazy (first request) and repeated after an exhaustion wait.MIN_EXHAUSTION_WAIT= 5s so stale reset timestamps can't busy-loop the quota endpoint, bounded bymax_wait_time, else raisestransient_error).max(budget_min_reserve, budget_reserve_fraction * limit), delays ofseconds_to_reset / total_remaining(capped at 10s) are injected for graceful degradation instead of a hard wall.Thread safety (new vs. the source-github implementation, required for the concurrent CDK):
RLock;_acquire_callreturns the token chosen under the lock and the request header is built from that value, so a request is always signed with the token whose counter was decremented (no TOCTOU with concurrent rotation). All sleeps occur outside the lock._refresh_lockso only one thread refreshes counters after an exhaustion wait.ModelToComponentFactory.create_rate_limited_multiple_token_authenticatorcaches instances keyed on the model definition, so all streams share one counter set (same idea as the sharedapi_budget).Example manifest usage (source-github):
Changes:
declarative_component_schema.yaml: newRateLimitedMultipleTokenAuthenticator,QuotaStatusSource,TokenQuotadefinitions; added toHttpRequester.authenticatorandSelectiveAuthenticator.authenticatorsunions (models regenerated; new classes applied on top of the checked-in file to avoid unrelated codegen churn).airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py: runtime implementation.model_to_component_factory.py:create_rate_limited_multiple_token_authenticatorwith token-string splitting, interpolation, and shared-instance caching.Requested by Daryna Ishchenko (darynaishchenko).
Link to Devin session: https://app.devin.ai/sessions/3ced8f6a11d44ed4abc517221294f767
Requested by: Daryna Ishchenko (@darynaishchenko)